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The evaluation design described in this paper 
cusses all significant aspects of program development and 
low-through for culturally disadvantaged children. Discussed 
cational objectives, instructional methods, implementation 
cedures, and outcomes. (CK) 
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Co'r.ponsa'Loi education has been recognized as a significant method 
Tor improving the academic. performance of? culturally disadvantaged 
children because it concentrates upon removing deficits which appear 
to hinder their further educational progress. However, the outcomes 



of many compensatory tduca.tn.on programs indicate that they do not 
produce the improvements in academic performance which are antici- 
pated by their educational objectives. One particular reason as to 
why expected outcomes are not attained in many of these programs 
is that they are not based upon a theoretical rationale which would 
make them acceptable in the sense required by scientific inquiry. 

It appears that educational programs which do not have a rationale 
derived from relevant educational and developmental theory, which 
explains why that given approach should be utilized with particular 
groups of culturally disadvantaged children, may eventuate in failure 
because the applied educational, objectives and methods were inappropriate 
for the group of children under investigation. However, in such 
situations, it is difficult to determine the cause(s) of failure 
because no theoretical, framework would have been involved to provide 
information concerning how the applied educational objectives and methods 



were expected to reduce specific types of deficits. On the other 
hand, a program which has a theoretical, basis in developmental and 
educational theory can provide useful information concerning why it 
should be applied to certain types of culturally disadvantaged children, 
and this Information can be utilised to explain either the success or 
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failure of a program 
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Programs which do rot have an explicit end valid hypo ihet leal 
g n.Fi to: '■ - can produce di!jiJj.r.sio^J:iono among cciiicr: col‘& b o c u us o *>hey 
(educators) are not provided vrith clear-cut guidelines concerning 
the populations to -which these programs should be applied, and 
therefore, they Kill hove difficulties in explaining why particular 
outcomes were obtained. These difficulties which are encountered 
in devising effective programs for culturally disadvantages children 
demonstrate the desirability for 'establishing a procedure by which 
to evaluate them, since diversity of conception without disciplined 
criteria as guides permits activity which is likely to result in 
waste or possibly undesirable outcomes* 

The evaluation design which is d -'scribed in this paper takes 
account of all significant aspects of program development and execution, 
including educational objectives, instructional methods, implemen- 
tation procedures _ and o\it comes, and the construction of this comprehen- 
save design involves a set of criteria that can be applied to evaluating 
these four phases of educational programs* In the preparations leading 
to the evaluative system as finally conceptualized, the j-o. ...ow-mg 
activities were undertaken: 

j, Educat io nal Ob j ective s : The selection of a scheme 

for arranging normal, developmental processes into 
a chronological sequence that was based upon the 
research and theory of Piaget and Inhelder (196?), 
and vihich was then used to: (l) identify the deficiencies 

of culturally disadvantaged children, and (2) organize 
these deficiencies into a taxonomy that follows the 
order of normal cognitive, social, and affective development. 
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II. 



Instructio nal Methods : The if* entif ication of instructional 

methods which have been utilized by exemplary com- 
pensatory education progi~ams, and the utilization of 
the Piaget and Inh elder scheme to construct a taxonomy 



of instructional methods that is arranged according to 
the areas of development they concentrate upon. 



III. Implemen ta t ion P rocedures : The specification of 

implementation procedures which have been utilized 
by some of the most outstanding programs for culturally 
disadvantaged children, and the organization of this 
information in terras of the characteristics of 
superior implementation procedures. 

IV. Outcomes: The identification of experimental designs 

vri. th high amounts of internal r’ld r terns > 

these designs being recommended by authorities in 
educational research, the isolation of standardized 
tests which have been utilized by exemplary education 
];rograms which are then arranged into a taxonomy that 
is parallel to the scheme of human development, and 
the specification of guidelines for testing cultural! 
disadvantaged children that are derived from the 
recommendations of experts in this area, of education. 
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Although a variety of educational programs have been developed 
for cultural^ y disadvantaged children, it appeal's thao many of tne^e 
pro^rnr.iG arc slirilnr in. teiTis of" ib-Gir exxeebivonoss in rexaooyiri^ 
particular types of deficits. The present evaluation design can be 
used as a frame of reference for deciding upon which one oi several 
different programs should be selected for application to a particular 
educational situation, since this design is constructed to facilitate 
the selection of programs that demonstrate the most promising methods 
for removing deficiencies in these children, tor example, this 
evaluation design will- enable educators to choose be owed language 
training approaches which appear to produce similar outcomes but 
which utilise apparently different educational methods. 

Illustrations of language training approaches which appear to 
be acceptable as to theoretical adequacy and as to final outcomes 
mid. be those of Berciter and Engelmonn (1966) and Blank said Solomon 
'(1969). In the former program, groups of about five disadvantaged 
children are given highly structured exercises in five areas of deficiency. 
These exercises consist of instruction in “verbatim repitition, 
yes-no questions, location tasks, statement production, and deduction 
problems. “ In contrast, the Blank and Solomon program has a 1-to-l 
pupil/tutor ratio, and the child acquires verbal facility through 
unstructured conversations with the tutor. As a result of those 
conversations, he learns to use verbal concepts such as “circle, 
across, ton and bottom' 1 to describe relationships among objects and 
events. The educator who wishes to decide upon which one of these 
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programs to util :i se must choose between tiro approaches thr o appear to 
be equally effective in terns of improving cognitive skill.s, sense 
both of these programs produce increases In IQ's of appr o::±mat ely 
twelve points, 'therefore, his decision as to which one to select should 
be based upon, considerations of all aspects of an educational program 



that are related to the quality and validity of its objectives, 
instructional methods, implementation procedures and outcomes. 

The arguments which exzist concerning sensori-notor versus 
conceptual training (Kohlberg, 1968$ Standi cr-Lavatolli, 1968) further 
illustrate a situation in which a c o ■ npr chens ive evaluation design 
will enable educators to make a rational, decision as to which one of 
these different approaches to adopt. Kohlberg argues that sensori-motor 
training, based upon the Ilontessori method ( 19610 , is one of the 
most effective methods for improving cognitive abilities. Therefore, 
the cultu rally disadvantaged children in his prograri learned to match 
•stimuli in terms of sensory qualities such as color or size, arrange 
stimuli along dimensions of light-dark and big- small., and differentiate 
between dimensions of loudness and pitch. In contrast, St endler-Lavat elli 
contends that training in certain rea.soning operations which have 



been, studied by Piaget (1969) will improve cognitive abilities. As a 
result of this position, Stendler-Lavatclli developed a program 
in which culturally disadvantaged children learned the reversibility, 
associativity, combinativity and identity operations. Ihc pres ent 
evaluation design will also enable educators to choose between thc.se 
two diffemt types of programs, since it provides criteria for 
selecting remedial, techniques that are most appropriate for promoting 
either sensori -motor or conceptual development. 

EKic „ 
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SUBSTANTIVE ACTIVITIES IiiVOLVKD tti CONSTRU STING THIS PltilGH 



The design for evaluating educational programs for culturally 
disadvantaged children was d ev cl o p c d in the following firmer in 
order to assure the adequacy of the theoretical basis of such programs: 
Reference System, fo r Ev aluating Ed ucatio nal Objectives* The Piagec. 
and Inholder Taxonomy of Hunan Development (196?) was used as a basis 
for describing the major cognitive, social and affective processes 
of normal children, and this particular taxonorqy was utilised because 
it is the most comprehensive one available. Subsequently, the literature 
on the deficiencies of culturally disadvantaged was studied, and this 
inf ormation was organized in a manner that was congruent with the 
psychological achcrae of human development, i.e., deficiencies associated 
with cognitive, social and affective processes were classif ied according 
to whether they developed in the sensori— motor or concrcoe operations 
period, and in terms of their specific relationship to the developmental 
processes described by Piaget and Xnhelder. Evaluative criteria for 
examining objectives were then derived from the taxonoiiiles of human 
development and deficiencies by using these taxonomies to identify 
important developmental processes which should be concentrated upon, 
and these criteria evaluate educational objectives in teams of their 
underlying theoretical rationale, developmental significance, appro- 
priateness for alleviating known deficiencies, comprehensiveness and 
the developmental sequence in which they are arranged. 
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Ref c'-cncc Sv.-.vevn for.' Evr nr.- J ^s t-ry ct :*• oral l Ed-ho ds «■ The instruction:! 

methods utilised by cxcr.pl ary c-duo aticri; 0 programs evaluated. by Rawkric3 0 c, ei 

(1968a) were identified end arranged into a Taxonomy of Instructional 

Method?. This taxonory was congruent with the scheme of human development. 

described by Piaget and Jnh elder (i.c., the instruction ?>3. methods were 

classified according to the areas of development that they concentrated 
won), and it included methods utilized by both preschool and primary 

school programs. The second group of evaluation criteria was based 

upon the f axonony of Instructional Methods, and the developed criteria 

specify that a current progress instructional methods should be similar 

to those described in the Taxonomy, comprehensive and arranged unto a 

sequence which is parallel to the sequence of human development. 

Reference System _for Evalu ating Im plementation 

evaluation studies of Hawkridge, ct al. ( 1968a, b) also provided 

information about the implementation procedures of exemplary programs, 
and this information was arranged into an inventory of the ••characteristics 

of superior implementation procedures" that described activities such 

as teacher training methods and the intensity of instruction associated 

vrith those programs. The criteria for the evaluation of implementation 

procedures wore set up to examine the six activities which were 

described as being the characteristics of superior implementation 

procedures. Thus, by utilizing these evaluation scales it can be determined 

whether a program was implemented in a manner congruent with acme 

of the most outstanding programs in the nation; and the following 

activities were evaluated: the amount of planning in selecting personnel 

and organizing a program, teacher training methods, ri-lcV<-~- 
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instruction end nr. tori :1s to objectives, intensity of treatment, 
siso of instruction-::! groups and amount of parental involvement. 

Refe rence fi rst on for liy-alu atlng O utco mes. Initially, two 
experimental designs described by Campbell and Stanley (1963) 
were identified that were highest in both internal and external 
validity. The * ! Solomon” and Poet-test Only Control Group’ 1 designs 
were then described in terms of their requirements for the random 
selection of subjects and the choice of separate control groups 
composed of both disadvantaged and middle- class children, and 
• further descriptions were given of the types of extraneous variables 
which are controlled by the use of these designs. 

This reference system also consisted of information concerning 
the use of appraisal techniques with disadvantaged children. Thus, a 
Taxonomy of Stand ^.rdiised Tests wo.s set up which classified repi escnfccv— 
tive assessment- devices according to the areas of development (parallel 
to Piaget and Inheldcr ) which they measure. These tests were selected 
from the exemplary programs evaluated by Ilawkridge, et al. (1968a, b), 
and the national evaluation of Head. St rat which includes approximately 
sixty tests (bTS, 1968). in addition, guidelines for using these tests 
with, disadvantaged children were also presented, and this part Oj. 
the reference system included information (from Petits eh, et al., 1961;) 
about methods for determining and improving upon the reliability of 
tests, and recommendations for upgrading the validity of test inter- 
pretation . 
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The final set of criteria (for outcomes) vas derived from the 
descriptions of superior experimental designs , the Taxono^ty of 
Standard! 7 , cd Tests and the guidelines for testing disadvantaged 
children. Initially, the evaluation scales for examining & program 1 s 
experimental design were constructed from the descriptions of superior 
experimental designs, and the scales for rating appraisal techniques 
wore established by reference to the information about selecting 
tests that are developmentally appropriate, and which have high 
amounts of reliability and validity. Thus, the evaluative instrument 
for this last phase of the design examines the internal, and external 
validity of a program’s experimental design, the relevance of the utilised 
tests to measuring outcomes predicted by objectives, the reliability 
and validity of tests which have been administered and the interpretation 

of test results* 

THE UTILITY OF THIS EVALUATION DESIGN 

If responsible critics (Bereiter & Engelmann, 1966; Jensen, 1969) 
of the literature of the culturally disadvantaged can be used for 
assumptions, this field of education is characterized by poor 
conceptualization, inadequate substance in the treatment processes. 
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and imprecise or irrelevant appraisals of progress or outcome. In 
those connections, the present design would appear to be significant 
in that it provides an organized and disciplined scheme for: 

a. C one eptualizin g programs > Information is provided concerning 
which educational objectives are most suitable for inclusion in 
educational programs, and this information is based upon the study 

of deficiencies which are associated with a comprehensive developmental 
scheme. 

b . Delim iting re search _to _thc_rios_t jSigni-ficant and theoretical ly 
integrable dimensions, var iables, etc. Only those educational methods 
which are based upon sound theoretical positions are considered. 

Therefore, this evaluation procedure encourages the use of methods 
which are firmly grounded in educational and developmental theory. 

c. Providing a st r ucture for org anicing information in this 
area of education . This inquiry provides a means of classifying 
different progrsxas in terms of educational goals, procedures, and outcomes. 
Hence, it aids in determining how programs are related to each other. 

d. Providing for mor e rig orous eva l uation . Procedures are developed 
for evaluating all phases of educational programs, ranging from an 
examination of the entering behavior of disadvantaged children 
through the testing procedures which are utilized to measure outcomes. 
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PROr.LElS Alii) IldUFb 



The most rigorous applications of this design require the use 
of rating scales which can reliably and validly measure how closely 
each of the four sets of criteria have been attained. Therefore, the 
problem of developing such scales will be treated in the following 
section. In addition, statements rail), be made concerning the educational 
qualifications of evaluators and the time at which the evaluation 
process should begin, since these factors will, significantly influence 
the evaluation process* The diseusrrion of those factors cone enti a ucs 
upon how they can affect evaluation in the public schools® 



Evalu ati on Sca les . This design is intended to comprise a necessary and 
sufficiently contained evaluation model and to be an advance over- 
available resources in the field. Further developments of this design 
for use by public school systems require the construction of psyeho- 
metrically reliable and valid scale standards in order to increase its 
sensitivity to the substantive differences between programs* Such scale 
standards can provide the necessary means for measuring how closely 
each criterion is fulfilled, if they include clear statements concerning 
the necessary characteristics which a program must have to receive a 
particular rating. However, this level of clarity can only be attained 
through field testing procedures that will yield information as to 



whether different evaluators give similar ratings when using the 
evaluation scales* Obviously, an instrument that is interpreted xn 
different ways must be modified to produce a high level of agreement, 
since this change will improve its reliability. Therefore, procedures 
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for deternirjinc the clarity of the 
involve te.k3.nc measures of inter-ra 



developed cvaluat ?. , on. scales rust 
ter reliabilities, and if these 



reliabilities are low, they can probably be increased by more 
precisely describing the characteristics that are required for 



assigning a particular rating, and by train.- •' valuators un oil th cy 
clearly understand the statements included, in each evaluation scale. 
The developed sca1.es must also have high amounts of external 



validity, since this measurement will indicate that they can correctly 
predict whether a program will be successful or fail, if it is applied 
to similar settings across the nation. However, this type of valioity 
can only be assessed after observing the outcomes of such applications. 



and then measuring the relationship between the original evaluation 



given to a program and the later results. Clearly, such measurements 
of external validity will indicate which scales should be eliminated 
from the evaluation scheme, and which ones can be used with the knowledge 



-that they enable the evaluator to make valid generalisations. 

In order to utilize this evaluation design as it is presently 
constructed., a five point rating scale can be utilized to determine how 
closely each criterion has been fulfilled, with a score of one 
representing the lowest possible rating. This rating device will only 
provide the most elementary information concerning a program's stiocess 
in attaining each of the four sets of criteria. But, it will also 
serve as- a beginning point for the empirical testing process essential 



-to the development ot a valid device*. 
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Q\i?:l.iricrdionu of Evaluators • Apparently, a direct relationship 
should exist between the educational background of the evaluator 
and his effectiveness in TvtiJ-i?.ing these particular criteria, i.c., 
an individual who has thoroughly studied developmental theories and 
research, experimental mewhods, and procedures for evaluating educational 
prop rams Kill probably apply this design more validly then mil 
persons without training in these areas. However, even those 
individuals who Hack specific training can use this design to improve 
their evaluation activities, if they study each of the four re tea once 
• system's carefully, and closely follow the procedures for deteradJiing 
whether each criterion has boon fulfilled* 

It appears that most evaluators of programs for disadvantaged 
children are trained in school administration rather than in areas 
such as child development and statistics because administrative 
personnel are usually responsible for overseeing and reporting upon 
the results of the extensive number of federally sponsored programs 
operating in the public schools. Therefore, it must be assumed that 
this evaluation scheme will be most frequently used by school .administrators 
who do not have the "ideal'* educational qualifications for thoroughly 
understanding and applying each of the four reference systems. In 
order for them to complete the most valid evaluation stucb.es, it is 
essential that they acquire some fundamental information about developmental 
theory, research methods and educational evaluation. In addition, 
further information can be obtained by studying the applications of this 
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design by professional 
ija these academic areas 
thorough qualifications 



evaluators who have extensive backgrounds 
, since these individuals vri.ll have the most 
for making valid applications. 



Proacti ve Ev aluat ion. It appears that the model presented here 

can be most effectively utilised to evaluate a program before 

it actually begins operating, ± . e . , the descriptions of objectives, 

instructional methods, implementation procedures and techniques 

for measuring outcomes which are presented in a project proposal can 

be cautiously examined to d.c'cnvidne whether major weaknesses exist 

in any of these phases of a program’s development. Thus, proactive 

evaluation may eliminate the dxfficxiltics which occur in modifying 

instructional methods, etc., based upon the results of formative evaluation 

studi.es because such changes are extremely difficult to make during 

the operational stage. 

In summary, it is proposed that rigorous scale standards be 
developed for each evaluative criterion, evaluators acquire some 
essential knovrledge of developmental theories .and research methodology 
before applying this design, and applications be made proactively 
in order to locate potential weaknesses which may cause a program to 
fail in achieving commendable objectives. Presumably, these improvements 
and uses will produce evaluative data which will significantly increase 
the quality of education for disadvantaged children. 
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